Added support for the database engine plugin system for extending sqlc with new databases (in addition to PostgreSQL, Dolphin (MySQL), sqlite) #4247

asmyasnikov · 2025-12-28T19:03:26Z

This PR resolve issue #4158

Database Engine Plugins

sqlc supports adding custom database backends through engine plugins. This allows you to use sqlc with databases that aren't natively supported (like YDB, Clickhouse, CockroachDB, or other SQL-compatible databases).

Overview

Engine plugins are external programs that implement the sqlc engine interface:

Process plugins (Go): Communicate via Protocol Buffers over stdin/stdout
WASM plugins (any language): Communicate via JSON over stdin/stdout

Compatibility Guarantee

For Go process plugins, compatibility is guaranteed at compile time:

import "github.com/sqlc-dev/sqlc/pkg/engine"

When you import this package:

If your plugin compiles successfully → it's compatible with this version of sqlc
If types change incompatibly → your plugin won't compile until you update it

The Protocol Buffer schema ensures binary compatibility. No version negotiation needed.

Configuration

sqlc.yaml

version: "2"

# Define engine plugins
engines:
  - name: somedb
    process:
      cmd: sqlc-engine-somedb
    env:
      - SOMEDB_CONNECTION_STRING

sql:
  - engine: somedb             # Use the SomeDB engine
    schema: "schema.sql"
    queries: "queries.sql"
    gen:
      go:
        package: db
        out: db

Configuration Options

Field	Description
`name`	Unique name for the engine (used in `sql[].engine`)
`process.cmd`	Command to run (must be in PATH or absolute path)
`wasm.url`	URL to download WASM module (`file://` or `https://`)
`wasm.sha256`	SHA256 checksum of the WASM module
`env`	Environment variables to pass to the plugin

Creating a Go Engine Plugin

1. Import the SDK

import "github.com/sqlc-dev/sqlc/pkg/engine"

2. Implement the Handler

package main

import (
    "github.com/sqlc-dev/sqlc/pkg/engine"
)

func main() {
    engine.Run(engine.Handler{
        PluginName:        "somedb",
        PluginVersion:     "1.0.0",
        Parse:             handleParse,
        GetCatalog:        handleGetCatalog,
        IsReservedKeyword: handleIsReservedKeyword,
        GetCommentSyntax:  handleGetCommentSyntax,
        GetDialect:        handleGetDialect,
    })
}

3. Implement Methods

Parse

Parses SQL text into statements with AST.

func handleParse(req *engine.ParseRequest) (*engine.ParseResponse, error) {
    sql := req.GetSql()
    // Parse SQL using your database's parser
    
    return &engine.ParseResponse{
        Statements: []*engine.Statement{
            {
                RawSql:       sql,
                StmtLocation: 0,
                StmtLen:      int32(len(sql)),
                AstJson:      astJSON, // AST encoded as JSON bytes
            },
        },
    }, nil
}

GetCatalog

Returns the initial catalog with built-in types and functions.

func handleGetCatalog(req *engine.GetCatalogRequest) (*engine.GetCatalogResponse, error) {
    return &engine.GetCatalogResponse{
        Catalog: &engine.Catalog{
            DefaultSchema: "public",
            Name:          "somedb",
            Schemas: []*engine.Schema{
                {
                    Name: "public",
                    Functions: []*engine.Function{
                        {Name: "now", ReturnType: &engine.DataType{Name: "timestamp"}},
                    },
                },
            },
        },
    }, nil
}

IsReservedKeyword

Checks if a string is a reserved keyword.

func handleIsReservedKeyword(req *engine.IsReservedKeywordRequest) (*engine.IsReservedKeywordResponse, error) {
    reserved := map[string]bool{
        "select": true, "from": true, "where": true,
    }
    return &engine.IsReservedKeywordResponse{
        IsReserved: reserved[strings.ToLower(req.GetKeyword())],
    }, nil
}

GetCommentSyntax

Returns supported SQL comment syntax.

func handleGetCommentSyntax(req *engine.GetCommentSyntaxRequest) (*engine.GetCommentSyntaxResponse, error) {
    return &engine.GetCommentSyntaxResponse{
        Dash:      true,  // -- comment
        SlashStar: true,  // /* comment */
        Hash:      false, // # comment
    }, nil
}

GetDialect

Returns SQL dialect information for formatting.

func handleGetDialect(req *engine.GetDialectRequest) (*engine.GetDialectResponse, error) {
    return &engine.GetDialectResponse{
        QuoteChar:   "`",             // Identifier quoting character
        ParamStyle:  "dollar",        // $1, $2, ...
        ParamPrefix: "$",             // Parameter prefix
        CastSyntax:  "cast_function", // CAST(x AS type) or "double_colon" for ::
    }, nil
}

4. Build and Install

go build -o sqlc-engine-somedb .
mv sqlc-engine-somedb /usr/local/bin/

Protocol

Process Plugins (Go)

Process plugins use Protocol Buffers for serialization:

sqlc → stdin (protobuf) → plugin → stdout (protobuf) → sqlc

The proto schema is published at buf.build/sqlc/sqlc in engine/engine.proto.

Methods are invoked as command-line arguments:

sqlc-engine-somedb parse        # stdin: ParseRequest, stdout: ParseResponse
sqlc-engine-somedb get_catalog  # stdin: GetCatalogRequest, stdout: GetCatalogResponse

WASM Plugins

WASM plugins use JSON for broader language compatibility:

sqlc → stdin (JSON) → wasm module → stdout (JSON) → sqlc

Full Example

See examples/plugin-based-codegen/ for a complete engine plugin implementation.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  sqlc generate                                                  │
│                                                                 │
│  1. Read sqlc.yaml                                              │
│  2. Find engine: somedb → look up in engines[]                    │
│  3. Run: sqlc-engine-somedb parse < schema.sql                    │
│  4. Get AST via protobuf on stdout                              │
│  5. Generate Go code                                            │
└─────────────────────────────────────────────────────────────────┘

Process Plugin Communication (Protobuf):

    sqlc                          sqlc-engine-somedb
    ────                          ────────────────
      │                                  │
      │──── spawn process ─────────────► │
      │     args: ["parse"]              │
      │                                  │
      │──── protobuf on stdin ─────────► │
      │     ParseRequest{sql: "..."}     │
      │                                  │
      │◄─── protobuf on stdout ───────── │
      │     ParseResponse{statements}    │
      │                                  │

.github/workflows/ci.yml

…c with new databases (in addition to PostgreSQL, Dolphin, sqlite)

protos/plugin/codegen.proto

asmyasnikov commented Dec 28, 2025

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

asmyasnikov added 10 commits January 10, 2026 15:07

Added support for the database engine plugin system for extending sql…

b6aa62d

…c with new databases (in addition to PostgreSQL, Dolphin, sqlite)

Fix of endtoend tests

5336821

added install plugin-based-codegen's

2b88994

remove tmp file

b1d156d

removed go.{mod,sum}

9f65d4f

SQLCDEBUG=processplugins=1

74b621f

Fix

cede5d3

Fix

15b240d

Apply suggestions from code review

0b3b165

revert Combine

6c5b9a6

asmyasnikov force-pushed the engine-plugin branch from fd040bc to 6c5b9a6 Compare January 10, 2026 12:07

.gitignore + README

7609ebc

asmyasnikov force-pushed the engine-plugin branch 2 times, most recently from a5131b5 to 7609ebc Compare January 26, 2026 17:28

simplified engine API

2c74313

asmyasnikov commented Jan 27, 2026

View reviewed changes

protos/plugin/codegen.proto Outdated Show resolved Hide resolved

asmyasnikov added 11 commits January 27, 2026 15:26

Apply suggestions from code review

88e6082

Delete protos/engine/engine_grpc.pb.go

f39ae4a

Delete protos/engine/engine.pb.go

18f5368

Delete pkg/plugin/sdk.go

8eaef3c

Delete pkg/engine/engine.pb.go

fbaf6ba

Delete pkg/plugin/codegen.pb.go

ce385ae

Delete examples/plugin-based-codegen/README.md

a024d3e

Delete examples/plugin-based-codegen/gen/rust/queries.rs

fbd5b43

docs

e6a730a

removed example

c8831c7

fix

6d5770f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for the database engine plugin system for extending sqlc with new databases (in addition to PostgreSQL, Dolphin (MySQL), sqlite) #4247

Added support for the database engine plugin system for extending sqlc with new databases (in addition to PostgreSQL, Dolphin (MySQL), sqlite) #4247

Uh oh!

asmyasnikov commented Dec 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Added support for the database engine plugin system for extending sqlc with new databases (in addition to PostgreSQL, Dolphin (MySQL), sqlite) #4247

Are you sure you want to change the base?

Added support for the database engine plugin system for extending sqlc with new databases (in addition to PostgreSQL, Dolphin (MySQL), sqlite) #4247

Uh oh!

Conversation

asmyasnikov commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Database Engine Plugins

Overview

Compatibility Guarantee

Configuration

sqlc.yaml

Configuration Options

Creating a Go Engine Plugin

1. Import the SDK

2. Implement the Handler

3. Implement Methods

Parse

GetCatalog

IsReservedKeyword

GetCommentSyntax

GetDialect

4. Build and Install

Protocol

Process Plugins (Go)

WASM Plugins

Full Example

Architecture

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

asmyasnikov commented Dec 28, 2025 •

edited

Loading